Search for: All records

We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the Gács-Körner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using a simple modification of a traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground-truth latent factors are known, we show that we can accurately quantify the common information between the random variables. 

We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the “redundant information”. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks. 

We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training. We show that the implicit regularization coming from training with Stochastic Gradient Descent with a high learning-rate and small batch size plays an important role in learning minimal sufficient representations for the task. In the process of arriving at a minimal sufficient representation, we find that the content of the representation changes dynamically during training. In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training, before being later discarded. In addition, we evaluate how perturbing the initial part of training impacts the learning dynamics and the resulting representations. We show these effects on both perceptual decision-making tasks inspired by neuroscience literature, as well as on standard image classification tasks. 

Electronic cigarettes (E-cigs) generate nicotine containing aerosols for inhalation and have emerged as a popular tobacco product among adolescents and young adults, yet little is known about their health effects due to their relatively recent introduction. Few studies have assessed the long-term effects of inhaling E-cigarette smoke or vapor. Here, we show that two months of E-cigarette exposure causes suppression of bone marrow hematopoietic stem and progenitor cells (HSPCs). Specifically, the common myeloid progenitors and granulocyte-macrophage progenitors were decreased in E-cig exposed animals compared to air exposed mice. Competitive reconstitution in bone marrow transplants was not affected by two months of E-cig exposure. When air and E-cig exposed mice were challenged with an inflammatory stimulus using lipopolysaccharide (LPS), competitive fitness between the two groups was not significantly different. However, mice transplanted with bone marrow from E-cigarette plus LPS exposed mice had elevated monocytes in their peripheral blood at five months post-transplant indicating a myeloid bias similar to responses of aged hematopoietic stem cells (HSC) to an acute inflammatory challenge. We also investigated whether E-cigarette exposure enhances the selective advantage of hematopoietic cells with myeloid malignancy associated mutations. E-cigarette exposure for one month slightly increased JAK2V617F mutant cells in peripheral blood but did not have an impact on TET2−/− cells. Altogether, our findings reveal that chronic E-cigarette exposure for two months alters the bone marrow HSPC populations but does not affect HSC reconstitution in primary transplants.